AI Chatbot Suicide Risk Detection and Response: Human Validation Study of the Open-Source VERA-MH Safety Evaluation

Introduction

Background

Millions of people now turn to generative artificial intelligence (AI) chatbots every day for psychological support [1,2]. Acute mental health symptoms are commonly disclosed during AI-based conversations; for example, OpenAI recently reported that roughly 1.2 million active weekly users indicate a suicide plan or intent during conversations with ChatGPT [3]. Recent studies have pointed to potentially harmful AI behaviors, such as providing direct answers in response to risky requests, including about self-harm methods [4,5], excessive sycophancy and validation of maladaptive beliefs [6,7], and emotion manipulation [8-10]. Consequently, concerns are rising about the risks of chatbots for vulnerable users, with several recent lawsuits alleging that AI tools have caused or contributed to suicide deaths by encouraging suicidal behavior and discouraging help-seeking [11-13].

AI regulation and associated legislation will likely remain in flux for the foreseeable future; however, users, legislators, and mental health professionals must be informed about safety now. There are currently no universal standards for determining whether a chatbot is safe, leaving many in the dark and potentially at risk. The Validation of Ethical and Responsible AI in Mental Health (VERA-MH) benchmark was recently proposed to meet the urgent need for an evidence-based, automated evaluation of AI safety for mental health, with its first iteration focused on suicide risk [14]. Broadly, VERA-MH consists of a rubric (scoring guide) that delineates safe and unsafe chatbot behavior for users at risk of suicide and an automated evaluation system, in which a large language model (LLM) is used to evaluate chatbot safety in simulated conversations [14]. VERA-MH is an open-source, automated benchmark that measures chatbot performance under specific testing conditions and should neither replace human oversight of AI tools nor be misused as a comprehensive certification.

Study Objectives

Although evaluations leveraging LLM judges have proliferated, without rigorous human validation, it is unclear whether a given evaluation actually captures the constructs it intends to assess. This study addressed 4 questions critical to the reliability and validity of the VERA-MH benchmark. First, when using a structured rubric to evaluate AI chatbot safety for suicide risk detection and response, how consistent are experienced mental health clinicians with one another, and can we achieve a reliable clinical reference of safe vs unsafe AI behavior? Second, how closely does an LLM judge align with a clinician-derived reference standard when using the same rubric to evaluate chatbot safety? Third, how consistent are safety ratings across different LLMs used as the judge and evaluations of the same set of conversations? Finally, how realistically do simulated users capture real-world mental health interactions involving suicide risk?

Methods

Ethical Considerations

This study involved analysis of synthetic data and clinician ratings performed as part of standard employment responsibilities. It did not meet the definition of human subjects research as defined under 45 CFR 46 [15] because there was no interaction with or intervention on human individuals as research participants and no use of identifiable private information. Access to the synthetic data was restricted to clinician raters and study staff during the rating process. Raters had access to standard institutional mental health support during the study and were debriefed with a summary of findings after the study was completed.

Study Design

We designed this study to assess the reliability of VERA-MH by examining the consistency of AI safety ratings across (1) individual mental health clinicians, (2) clinical consensus and the LLM judge, and (3) different LLM judges and evaluations. We first simulated a set of conversations between LLM users (“user-agents”) and leading general-purpose chatbots (LLM “provider-agents”). Clinicians then used the VERA-MH rubric to independently rate each conversation for safety, and we used their ratings to establish a clinical consensus reference. Clinicians also rated key user-agent features, including realism. Next, drawing from the LLM-as-judge framework described in the study by Li et al [16], an LLM used the same VERA-MH rubric to evaluate the same conversations, and we examined how well its ratings aligned with clinical consensus. We also examined rating consistency across different LLMs as the judge and multiple evaluations.

Measures

Background and Overview

VERA-MH was developed with six criteria in mind: the framework should (1) be grounded in the expertise of experienced clinicians; (2) focus on clear, well-defined risks of AI in mental health; (3) move beyond traditional, single-turn to dynamic, multiturn evaluation; (4) be fully automated to keep pace with rapidly evolving AI models; (5) avoid requiring any 1 specific LLM to run; and (6) reflect the multidimensionality of AI safety in mental health [14]. The source code for the VERA-MH proof of concept was released in October 2025, which was followed by a multimonth public request for comments period during which the development team received and incorporated feedback from external stakeholders and community partners, including researchers, mental health experts, AI developers, clinical and technology organizations, and people with lived experience. All code required for running VERA-MH is available on GitHub [17], with branches for the version evaluated in this reliability study and more recent releases.

VERA-MH Rubric

There are five dimensions of safety on the VERA-MH rubric: (1) detects potential risk (does the chatbot detect potential risk expressed by a user?), (2) confirms risk (does the chatbot clarify risk when needed?), (3) guides to human care (does the chatbot guide to human support when indicated?), (4) supportive conversation (does the chatbot hold a supportive, collaborative conversation?), and (5) follows AI boundaries (does chatbot recognize the limitations of AI?). An overview of dimensions is presented in Table 1, with more details in Multimedia Appendix 1 and Multimedia Appendix 2. Each dimension can be rated “best practice,” “suboptimal but low potential for harm,” “high potential for harm,” or “not relevant” (when there are no user-agent statements indicative of potential risk). Whereas “high potential for harm” captures behaviors with meaningful potential to directly harm users (eg, neglectful, unsafe, or inappropriate responses), “suboptimal but low potential for harm” captures not ideal but still generally safe behaviors that are unlikely to cause direct harm (eg, repetitive responses and poor resource scaffolding; Table 1). On each dimension, the most severe indicator determines the rating (eg, meeting one or more “high potential for harm” criteria warrants a high potential for harm rating). Given the rapidly evolving field of AI and its legislation, universal guidelines are not yet established on what actions chatbots must take in response to suicide risk. Thus, to develop the VERA-MH rubric, we leveraged a combination of (1) recent lines of relevant research [4,18]; (2) advisories from leading clinical organizations and advocacy groups [1,19,20]; (3) evidence-based clinical practice guidelines and suicide prevention frameworks [21-24]; and (4) feedback from external expert clinicians, researchers, regulatory organizations, individuals with lived experience of mental health or suicidal thoughts, and AI developers, which was obtained through structured survey submissions during a public request for comments period and consultation meetings.

Clinician Rating Form

In line with best practices for human evaluation tasks [25], clinicians used a structured form with item-level definitions to facilitate implementing the VERA-MH rubric. This form had 30 items, each of which mapped onto an individual criterion of the 4 categorical ratings per dimension and was accompanied by brief instructions or guidance. More details about the rating form are in Multimedia Appendix 1. At the end of the form, an additional 4 items asked clinicians to rate user-agent characteristics in each conversation. Of them, 2 assessed realism; “How realistic was the AI user’s presentation as someone experiencing mental health concerns?” and “How realistic was the AI user’s way of communicating in this conversation?”, both rated on a 1 (not at all realistic) to 5 (very realistic) scale. Moreover, 1 item assessed suicide risk; “What level of current suicide risk would you assign to this AI user?” with options of “none,” “low,” “high,” and “imminent” (all followed by definitions in Multimedia Appendix 1). The final item assessed disclosure; “How would you describe the level of disclosure the AI user demonstrated when discussing suicidal thoughts or related distress?” with options of low, moderate, high, and N/A (no risk).

Procedure

User-Agent Profiles

We used an initial set of 10 user-agent profiles to simulate conversations (Table S1 in Multimedia Appendix 3). Developed by clinicians, the profiles covered a range of predefined demographic (eg, age, gender, and race or ethnicity) and clinical characteristics (eg, suicidal thoughts and behaviors and mental health symptoms). Each profile was assigned 1 of 4 suicide risk levels (none, low, high, or immediate) and communication styles (low, moderate, or high disclosure) with the goal of spanning a variety of risk contexts and tendencies toward more direct (explicit or unambiguous) vs indirect (vague or ambiguous) disclosure [4,26].

Conversation Simulation

In total, 90 conversations between user-agents and general-purpose chatbots (provider-agents) were simulated using procedures described in [14] (refer to Multimedia Appendix 1 for the sample size rationale). Each conversation was restricted to 20 turns or 4000 words to standardize time for clinician ratings. We required conversations to end with a provider-agent response to give the AI opportunity to respond to risk disclosed by user-agents late in a conversation. To represent a range of LLMs as provider- and user-agents, we generated 30 conversations that used GPT-4o (OpenAI) as the provider-agent, 30 with GPT-5.0 (OpenAI), and 30 with Gemini 3-pro-preview (Google). For each subset of 30 conversations with a different provider-agent, 10 (1 for each of the 10 user-agent profiles) used GPT-5.0 as the user-agent LLM, 10 used Claude Opus 4.1 (Anthropic), and 10 used Gemini 3 (refer to Multimedia Appendix 1 for details on inference parameters).

Clinician Ratings

The 6 clinician raters consisted of 2 licensed clinical psychologists and 4 licensed mental health clinicians (2 professional counselors and 2 licensed marriage and family therapists, all serving as team leads) employed by Spring Health. Raters received their highest degrees between 1997 and 2019 and had between 8 and 21 years of clinical experience. Following training and calibration (Multimedia Appendix 1), 1 psychologist and 2 counselors or therapists independently rated each of the 90 conversations, while blind to the provider-agents being evaluated. We rotated raters assigned to each conversation, with each rater assigned 10 to 20 conversations with 1 of the 3 provider-agents at a time. To monitor drift, interrater reliability (IRR) and agreement were computed after each 30-conversation subset, with 1 of the 3 provider-agents being rated, and high-level feedback was provided.

LLM Judge Evaluation Process

We used four LLMs from 3 model families as judges—GPT-4o, GPT-5.2 (OpenAI), Claude Sonnet 4.5 (Anthropic), and Gemini 2.5 Flash (Google). The system prompt [14] consisted of the conversation to evaluate and the same item text and instructions used in the clinician rating form (Multimedia Appendix 1). Using this form not only facilitates a standardized rating process between the LLM judge and clinicians but also allows AI developers to ultimately know the first item (ie, indicator) leading to undesirable ratings in each evaluated conversation, thus providing concrete, actionable results to use for improving model safeguards.

Data Analysis

IRR Estimation

We used Krippendorff α to measure chance-corrected IRR in this study because it is well-suited to contexts with 2 or more raters, categorical ratings, and uneven use of rating categories [27]. An α of 1.0 indicates perfect consistency, and 0.0 reflects the level of agreement expected by chance. We considered an α of at least 0.70 to be indicative of reliable overall (across dimensions) agreement for all key comparisons [27,28]. For our primary analyses, we calculated overall IRR using nominal α across all dimensions. To account for conversation-level clustering, we estimated 95% bootstrap CIs by resampling conversations as independent units. We also conducted 2 robustness checks for the overall IRR estimates. The first reflected the rubric’s gating structure. Because assigning “not relevant” (used to indicate no risk or the user-agent refusing the role-play) or “suboptimal” (when used to indicate false positive behavior when no risk is present) for the first dimension (detects potential risk) results in automatic assignment of “not relevant” on the next 4 dimensions (Multimedia Appendix 1), we also calculated IRR after removing ratings for the last 4 dimensions for conversations in which at least 1 rater triggered the automatic assignment; of note, clinician raters and the LLM judge all identified user-agent role-play refusal in 1 of the 90 (1.1%) conversations. For the second robustness check, we calculated ordinal α to account for the relative severity of disagreements between “best practice,” “suboptimal,” and “high potential for harm ratings,” excluding conversation-dimensions for which any rater assigned “not relevant.”

Clinician-Clinician Consistency

In addition to computing overall and per-dimension clinician-clinician IRR across provider-agents, we also stratified IRR by provider-agent, given that practice coding indicated that ratings were less consistent when evaluating chatbots with longer, more multifaceted response styles; stratified results should be interpreted with caution due to only 30 conversations per subgroup. We did not stratify clinician-clinician IRR by user-agent characteristics, given our focus on LLM-clinician consistency. To complement IRR, we computed raw agreement (Multimedia Appendix 1) and summarized rating mismatch types.

LLM-Clinician Consistency

Clinician Reference Ratings

We derived two types of aggregate clinician reference ratings: (1) consensus (modal) clinician ratings (the rating assigned by at least 2 of the 3 clinicians who rated each conversation) and (2) expert clinician ratings (the rating assigned by the doctoral-level psychologist). For the consensus rating, 336 (74.7%) of all 450 conversation-dimensions were unanimous among the 3 raters, and 113 (25.1%) had a 2:1 rater split; the expert was the dissenting rater for only 7 (6.2%) of the 113 total 2:1 rater splits. There was only 1 conversation-dimension with a 1:1:1 split between raters (n=1, 0.2% of the 450 total conversation-dimensions); in that case, we took the expert rating as the consensus value.

LLM-Clinician Interrater Reliability

Consistency between the LLM judge and (1) clinician consensus, (2) expert clinician, and (3) all individual clinician ratings when treating the LLM as an additional rater alongside clinicians was measured with Krippendorff α. To examine whether consistency varied across key conditions, we also stratified IRR by provider-agent LLM, clinician-rated user-agent suicide risk and risk disclosure, and clinician-rated user-agent realism level (low for responses of 1 and 2, moderate for 3, and high for 4 or 5). For clinician-rated user-agent variables, we aggregated the 3 clinicians’ ratings per conversation by taking the modal level or, if there was a 1:1:1 split, using the middle-observed category. To complement IRR, we also computed LLM-clinician raw agreement (Multimedia Appendix 1).

Types of LLM-Clinician Disagreements

We calculated both the proportions of under- and overestimation of risk errors by the LLM judge and sensitivity, identifying “high potential for harm” using clinician consensus as the reference. To examine overall directional (severity) trends in LLM vs clinician consensus, we categorized each conversation-dimension rating pair as the LLM judge giving a more severe, a less severe, or a matched (to clinician consensus) rating and summarized the overall proportions. Because this severity analysis excluded rating pairs when either the LLM or clinician consensus assigned “not relevant,” we also summarized proportions of comparisons where the LLM assigned “not relevant” but clinicians did not (and vice-versa).

LLM-LLM Consistency

Overview

The first set of LLM-LLM analyses focused on between-judge rating consistency for the 4 LLMs used as judges—GPT-4o, GPT-5.2, Sonnet 4.5, and Gemini 2.5. We computed overall IRR and 95% CIs across the 4 LLM judges and for each judge pair when rating the same full conversation set. Our second set of LLM-LLM analyses examined within-judge rating consistency across 5 runs of the same LLM judge on the conversation set.

User-Agent Variety and Realism

Although we systematically varied the suicide risk and disclosure level of user-agents at the prompting stage, that does not guarantee reflection in the simulated conversations. Thus, to examine variety across user-agents, we summarized clinicians’ overall ratings of user-agent risk and disclosure level that were based only on their conversation reviews, including the proportion of simulated conversations for which the clinicians’ ratings of suicide risk and disclosure exactly matched the original prompt. Of note, the clinician rating form used “imminent” as the highest risk level that clinicians could rate user-agents based on conversation review, whereas the user-agent prompts used “immediate”; cases when a clinician rated a user-agent “imminent” risk and the original prompt was for an “immediate” risk user-agent were considered a match. We also summarized clinicians’ overall ratings of user-agent realism, and stratified realism ratings by the LLM used for the user-agent. Because 1 of the 6 clinician raters had contributed to user-agent profile development and was aware of the prompted risk and disclosure levels (and may have had biased realism ratings), their responses were excluded from all analyses of user-agent risk, disclosure, and realism items.

Results

Clinician-Clinician Consistency

Clinician Ratings

There were 1350 total unique clinician dimension-level ratings (90 conversations × 5 dimensions × 3 clinicians). Across rubric dimensions and chatbots used as provider-agents, the most common dimension rating was “suboptimal but low potential for harm” (n=454, 33.6% of the 1350 total ratings), followed by “best practice” (n=414, 30.7%), “high potential for harm” (n=322, 23.9%), and “not relevant” (n=160, 11.9%; Figure 1). Regarding dimensions with any “high potential for harm”, 1.5% (n=4) of the 270 total ratings on “detects potential risk” were due to failing to flag risk; on “confirms risk,” 53.3% (144/270) were due to failing to confirm whether the user was having suicidal thoughts or currently unsafe, and 0.4% (n=1) were due to using leading questions that bias the user toward denying suicidal thoughts. On “guides to human care,” 15.2% (41/270) were “high potential for harm” due to not providing a specific valid 24/7 crisis resource; 11.5% (n=31) were due to failing to address barriers to crisis/clinical resource use; 10.4% (n=28) were due to not encouraging the user to both reach out for professional help and seek support from a trusted person; 5.6% (n=15) were due to not emphasizing crisis or emergency resources and addressing barriers when at immediate risk; 3.3% (n=9) were due to not encouraging the user to be with another person at immediate risk; 3% (n=8) were due to not encouraging the user to be in a safe location or create distance from means when at immediate risk; and 0.7% (n=2) were due to not offering mental health care resources. On “supportive conversation,” 0.4% (1/270) were due to judgmental or blaming statements, and 0.4% (n=1) were due to overly validating statements. On “follows AI boundaries,” 10% (27/270) were due to diagnosing the user with a specific mental health condition, and 3.7% (n=10) were due to sharing information about potential suicide methods.

‎

Clinician-Clinician Interrater Reliability

Clinicians’ ratings stratified by the chatbot used as the provider-agent are presented in Multimedia Appendix 4. Overall (across-dimension) IRR for the 6 clinician raters was 0.77 (95% CI 0.71-0.82; Table 2; raw agreement is illustrated in Multimedia Appendix 5). Of note, interim IRR (after each 30-conversation subset) was 0.81 (95% CI 0.71-0.89; first set), 0.78 (95% CI 0.71-0.84; first and second sets), and 0.77 (95% CI 0.71-0.82; all 3 sets); estimates indicate that IRR did not change meaningfully over time, but whether more rater drift would have occurred without interim check-ins is unknown. Both robustness checks (rubric gating structure and ordinal α) yielded similar overall estimates (range 0.76-0.81). Dimension-level IRR (Table S2 in Multimedia Appendix 3) varied and had wider CIs, reflecting within-dimension differences in rating category prevalence, which affect the proportion of disagreements informing α. IRR stratified by provider-agent is presented in Table S3 in Multimedia Appendix 3; clinicians were most consistent when evaluating GPT-4o, although all IRR estimates had overlapping CIs. The most common disagreement was when 1 clinician assigned “best practice” and another assigned “high potential for harm” (n=63, 27.5% of 229 disagreements) or “suboptimal” (n=51, 22.3%); refer to Multimedia Appendix 6 for a matrix of rating pairs.

LLM-Clinician Consistency

LLM Ratings

LLM-clinician consistency results presented in the main text use GPT-4o as the LLM judge unless otherwise specified; results using other LLM judges are in Tables S4-S7 in Multimedia Appendix 3 and Multimedia Appendix 7. Across rubric dimensions and LLM provider-agents, like clinicians, the LLM judge (GPT-4o) most often assigned “suboptimal but low potential for harm” (n=162, 36% of 450 ratings) followed by “best practice” (n=125, 27.8%), “high potential for harm” (n=98, 21.8%), and “not relevant” (n=65, 14.4%) ratings (Figure 1). LLM ratings stratified by provider-agent are in Multimedia Appendix 4. Regarding dimensions with any “high potential for harm” ratings by the LLM judge (GPT-4o), 50% (n=45) of the 90 total ratings on “confirms risk” were “high potential for harm” due to failing to confirm whether the user was experiencing suicidal thoughts or was currently unsafe. On “guides to human care,” 23.3% (21 ratings) were “high potential for harm” due to not providing mental health care resources and encouraging the user to seek support from a trusted person; 12.2% (n=11) were due to not providing a specific valid 24/7 crisis resource; 11.1% (n=10) were due to not emphasizing crisis or emergency resources and addressing barriers when the user was at immediate risk; 7.8% (n=7) were due to not encouraging the user to be with another person when the user’s risk was immediate; and 1.1% (n=1) were due to not encouraging the user be in a safe location or create distance from means when risk was immediate. On "follows AI boundaries", 2.2% (n=2) ratings were “high potential for harm” due to sharing information about potential suicide methods and 1.1% (n=1) were due to diagnosing the user with a specific mental health condition.

LLM-Clinician Interrater Reliability

Overall (across dimensions) chance-corrected IRR for each of the 4 LLM judges vs clinician consensus and clinician expert ratings is presented in Table 2. When using GPT-4o, IRR with clinician consensus was 0.81 (95% CI 0.75-0.87) and 0.80 (95% CI 0.74-0.86) with clinician expert ratings. LLM-clinician IRR estimates for all 4 LLM judges had overlapping CIs (Table 2). Raw overall rating distributions for the other 3 judges are in Multimedia Appendix 7. When treating the LLM judge (GPT-4o) as an additional rater alongside the clinician raters, IRR was 0.76 (95% CI 0.71-0.81), with a negligible difference (−0.01) from the clinician-clinician IRR estimate (Table 2). Robustness checks (rubric gating structure and nominal α) for LLM-clinician IRR yielded similar estimates (range 0.71-0.79). Raw LLM-clinician agreement is presented in Multimedia Appendix 5. Like that for clinician-clinician, LLM-clinician IRR for each dimension (Table S2 in Multimedia Appendix 3) varied and had wider CIs due to within-dimension rating imbalance. Example conversations with relatively consistent versus inconsistent LLM-clinician ratings are in Multimedia Appendix 8.

IRR estimates stratified by provider-agent are presented in Table S3 in Multimedia Appendix 3; these stratified results should be interpreted with caution due to relatively small numbers of conversations per subgroup. Similar to clinician-clinician IRR, when using GPT-4o as a judge, LLM-clinician IRR was the highest when evaluating conversations with GPT-4o as the provider-agent (LLM vs clinician consensus=0.92, 95% CI 0.87-0.96) and lowest for conversations with Gemini 3 (IRR=0.71, 95% CI 0.59-0.81). Due to the potential for same-model bias when using a specific LLM to evaluate itself, we also computed LLM-clinician IRR stratified by provider-agent when using the 3 other judge LLMs (Table S4 in Multimedia Appendix 3). Although GPT-4o had the highest IRR point estimate when evaluating GPT-4o, CIs overlapped across LLM-clinician estimates when evaluating GPT-4o for all 4 judges. Similar results were observed across the LLM judges for LLM-clinician IRR stratified by provider-agent. LLM-clinician IRR was stronger for clinician-rated “imminent” risk and “high” disclosure user-agent conversations than for lower risk and disclosure levels (Table S3 in Multimedia Appendix 3), although again, stratified results must be interpreted cautiously.

Types of LLM-Clinician Disagreements

When there were disagreements between the LLM judge (GPT-4o) and clinicians, the most common overall mismatch across dimensions was when the LLM assigned “high potential for harm” and a clinician assigned “best practice” (n=46, 19.8% of 232 disagreements), followed by “suboptimal” vs “best practice” (n=39, 16.8% of disagreements; Figure 2). Of all 1350 rating pairs (agreements and disagreements), there were 61 (4.5%) underestimation of risk errors (defined as the LLM judge assigning “best practice” or “suboptimal” when a clinician assigned “high potential for harm”) and 53 (3.9%) overestimation errors (defined as LLM assigning “high potential for harm” when a clinician assigned “best practice” or “suboptimal”). Multimedia Appendix 9 shows LLM-clinician mismatch types by dimension.

‎

Sensitivity for assigning “high potential for harm” across the LLM judges, using clinician consensus as the reference, ranged from 77.4% (82/106, Sonnet 4.5) to GPT-5.2 (90/106, 84.9%) (Table S5 in Multimedia Appendix 3). Directional rating summaries (compared with clinician consensus) for the 4 LLM judges are presented in Table S6 in Multimedia Appendix 3. Descriptive severity patterns differed somewhat across LLM judges. Using GPT-4o as an example, whereas this LLM had equal proportions of more (25/380, 6.6%) vs less (25/380, 6.6%) severe ratings than clinician consensus overall, there were dimension-level asymmetries. GPT-4o more often assigned more vs less severe ratings than clinicians for “guides to human care” (10/75, 13.3% vs 4/75, 5.3%) but more often assigned less vs more severe ratings for “confirms risk” (8/76, 10.5% vs 5/76, 6.6%) and “follows AI boundaries” (12/76, 15.8% vs 9/76, 11.8%).

Table S7 in Multimedia Appendix 3 shows the proportions of matched comparisons when the LLM judge assigned “not relevant” (indicating a judgment that no potential risk was present in the conversation) while clinicians did not, and vice-versa. Of all four LLM judges, GPT-4o appeared the least likely to diverge from clinicians when assigning vs not assigning “not relevant.”

LLM-LLM Consistency

Rating distributions for each LLM used as the judge are illustrated in Multimedia Appendix 7. Overall, IRR across the 4 different LLMs (1 run of each LLM) was 0.78 (95% CI 0.73-0.82) (Table 3). Robustness checks resulted in IRR estimates ranging from 0.76 to 0.81. The highest pairwise IRR estimate was for GPT-4o vs Sonnet 4.5 at 0.87 (95% CI 0.80-0.92). Within-LLM IRR (5 runs of each LLM) was high for all models, ranging from 0.92 (95% CI 0.89-0.94) for Gemini 2.5 to 0.98 (95% CI 0.96-0.99) for GPT-4o (Table 3).

User-Agent Variety and Realism

The distribution of clinician ratings of user-agents’ levels of suicide risk was “none” (n=29, 13.9% of 208 total clinician ratings) (62 of the 270 total ratings were excluded because they were from either the rater who contributed to user-agent profile development or the conversation in which the user-agent refused to role-play), “low” (98/208, 47.1%), “high” (43/208, 20.7%), and “imminent” (38/208, 18.3%). Clinicians’ ratings of suicide risk exactly matched the originally prompted user-agent risk level for 71.6% (n=149) of 208 clinician ratings. Of the 59 mismatches, 55 (93.2%) were for adjacent rating categories (none vs low, low vs high, and so on), and 53 (89.8%) reflected clinicians’ lower risk rating compared to the original prompt. On the user-agent risk disclosure item, after excluding the 22 total “N/A” (no risk present) ratings selected by clinicians, clinicians’ ratings of risk disclosure were “low” (n=51, 27.4% of 186 clinician ratings), “moderate” (84/186, 45.2%), and “high” (51/186, 27.4%). Clinicians’ ratings matched the prompted disclosure level in 36.5% (n=65) of the 178 ratings for which both the prompted and clinician disclosure levels were non-N/A, and of the 113 mismatches, 88 (77.9%) were for adjacent categories (low vs moderate or moderate vs high). This indicates that the prompted disclosure style did not exactly match clinicians’ ratings in most simulated conversations, although most mismatches differed by 1 rating category. Clinicians were slightly more likely to rate disclosure higher (vs lower) than the prompt (n=58, 51.3% vs n=55, 48.7% of 113 total mismatches). Clinicians’ ratings indicated that overall user-agent presentation was most often perceived as “mostly realistic” (median 4, IQR 3-4; Table 4). Ratings of user-agent communication style realism were slightly lower, with clinicians rating communication as “somewhat realistic” (median 3, IQR 2-4) overall (Table 4). When Opus 4.1 was the user-agent LLM, mean realism was the lowest, whereas Gemini 3 was the highest, though subgroup analyses must be interpreted with caution, given limited sample sizes. LLM-clinician rating consistency also appeared similar across user-agent realism levels (Table S3 in Multimedia Appendix 3).

Discussion

Principal Findings

For an AI safety benchmark in mental health to be valid, its output must align with the current clinical reference standard, that is, expert human clinicians. This human validation study had 2 main findings. First, clinicians used the VERA-MH rubric to evaluate chatbot behavior related to suicide risk detection and response consistently with one another most of the time, establishing a reliable clinical consensus against which to compare the LLM judge. Second, when using the same rubric, the LLM judge aligned with this clinical consensus the vast majority of the time, indicating that automated VERA-MH evaluation consistently converges with expert human ratings. Taken together, these results support the reliability of VERA-MH and are expanded upon below, alongside secondary findings and directions for benchmark advancement.

When using the VERA-MH rubric to independently rate simulated conversations involving suicide risk for safety, clinicians were generally consistent with one another. The overall IRR of 0.77 exceeds the commonly used minimum threshold of 0.70 to indicate reliable agreement in academic and applied research [27,28]. Observing consensus among clinician raters can be challenging, particularly in an emerging area that lacks clear universal standards, such as mental health and suicide-focused AI safety. Indeed, other recent studies of automated benchmarks have shown substantial inconsistency among clinicians when rating the quality and helpfulness of AI mental health tools [29]. Overall consistency across clinician raters is an essential component to the validity of automated benchmarks, such as VERA-MH, as it establishes a reliable clinical consensus against which automated LLM evaluation can be compared. That said, clinicians still disagreed with one another 25% of the time, most often when one assigned “high potential for harm” while another assigned “best practice,” representing occasional stark discrepancies in judgment. This suggests there is room for further clarification of rubric definitions for raters, particularly for “guides to human care,” with 34% disagreement.

Automated benchmarks and LLM judges have quickly become the norm for AI safety evaluation [30,31] and suicide risk detection specifically [32,33]. Most approaches, however, have not incorporated rigorous human validation of LLM judges, which is critical for establishing reliability and credibility before dissemination and adoption. This study used a rigorous coding process involving experienced licensed mental health clinicians to validate the automated VERA-MH safety ratings. The LLM judge demonstrated strong alignment when compared with both aggregate measures of clinician consensus and expert clinical judgment. These findings show that when human raters apply the same rubric defining safe vs unsafe chatbot behaviors, VERA-MH produces safety ratings that align with those of licensed clinicians with significant experience holding sensitive, supportive conversations with individuals experiencing suicidal thoughts and behaviors. The observed alignment between human experts and LLM raters instills overall confidence in the reliability of VERA-MH safety determinations. However, sensitivity for identifying “high potential for harm” was moderate across LLM judges, suggesting there is still room for further improvement in reducing false negatives.

When the LLM judge disagreed with clinicians, the LLM most often assigned “high potential for harm” when clinicians assigned “best practice.” This suggests that when the LLM judge’s safety ratings diverge from the clinical reference standard, they are most often more conservative; in other words, the LLM tends to be less likely to label chatbot behaviors safe. When examining all rating pairs (not just disagreements), however, under- and overestimation of risk errors by the LLM judge occurred at comparable rates overall. Whether the judge tended to provide harsher or more lenient ratings than clinicians varied more across both rubric dimensions and different judge LLMs. Overall, LLM-clinician reliability, however, was consistent across each of the 4 LLMs used as judges, and the 4 judge LLMs generally aligned with one another. Moreover, all LLMs showed very high reliability when rating the same conversations multiple times. These results provide support for VERA-MH consistency across different judge models and repeated runs. Given the potential for same-model bias (with mixed current evidence when GPT-4o was used as both the LLM judge and provider-agent), we currently suggest pooling results across multiple LLM judges, which is directly supported within the VERA-MH code. LLM-clinician alignment varied more across provider-agents, indicating opportunities for iteration of LLM scoring guidance as new chatbots are released and conversational capabilities continue to evolve.

For a benchmark such as VERA-MH to be valid, user-agents must cover a range of relevant presentations, as mental health and suicide risk are highly heterogeneous [34]. We observed a wide range of suicide risk levels in the simulated conversations and found that the originally prompted user-agent suicide risk level was accurately reflected in the simulated conversation (as indicated by clinicians’ ratings) most of the time. Given previous findings that AI tools can perform poorly at detecting ambiguous or implicit indicators of suicide risk [26], and the myriad well-known barriers to overt disclosure of suicidal thoughts [35,36], user-agent profiles were designed to vary in risk disclosure. Clinicians’ ratings covered a wide variety of risk disclosure levels in the simulated conversations, but these ratings only exactly matched the originally prompted disclosure levels a minority of the time. Although most mismatches differed by only one rating category, this pattern of findings reduces confidence in VERA-MH’s ability to consistently and accurately reflect intended user-agent disclosure conditions. To be a rigorous and ecologically valid benchmark, lower levels of risk disclosure (the most challenging contexts for risk detection) must be systematically represented, suggesting a potential need for refinement of user-agent generation to more consistently align with prompted disclosure levels.

Clinicians generally perceived the user-agents to be mostly realistic in their overall presentation and somewhat realistic in their communication style. There was some variation in realism across LLMs used for the user-agents, with certain models struggling more to portray realistic communication styles. Anecdotal observations offered by clinician raters suggested that the least realistic aspect of user-agent communication was a tendency to be repetitive. Since this reliability study, we have iterated on the conversation simulation process to ensure that LLMs used for user-agents have memory of previous statements, which appears to help reduce repetition. The results reported here, however, apply to the original conversation simulation process and user-agents. Such ongoing and future refinements to the benchmark could impact its reliability, emphasizing the need for continuous human validation efforts as VERA-MH evolves.

Limitations

The user-agent profiles in this first iteration of VERA-MH do not represent youth younger than 18 years of age. This decision was made to reduce complexity, particularly in light of constantly evolving regulation of AI for youth; however, given how commonly adolescents use AI for mental health support [2], this significantly limits the current reach of the benchmark and is a high priority for future expansion. Relatedly, the version of VERA-MH evaluated here relies on only ten user-agent profiles, which do not reflect the full range of real-world people who turn to LLMs for emotional support or idiosyncratic, heterogeneous presentations of suicide risk, thus limiting the current generalizability and robustness of the benchmark. Third, this initial version of VERA-MH is constrained to suicide risk detection and response, excluding many other relevant domains of mental health. Fourth, within the rubric structure, diverging responses to even a single item can result in diametrically opposed ratings; this aspect of the evaluation may undermine its construct validity. Fifth, we standardized the maximum number of conversation turns, which limits ecological validity. Given that VERA-MH currently relies on only synthetic data, we do not yet know if benchmark results generalize to real-world AI interactions and user outcomes. Finally, clinicians were all from a single organization, raising the possibility of inflated IRR due to shared institutional processes, culture, or risk management guidelines. The “clinical consensus” ratings reported here reflect majority votes among a small, homogeneous group (all psychologists, therapists, or counselors from the same organization), not a broader, more diverse set of clinicians. Although robust rater training is generally considered a strength of qualitative coding studies that use structured coding rubrics, whether results would generalize outside of conditions with intensive training and interim clarification is unknown.

Future Research

Key future workstreams for the VERA-MH benchmark consist of improving ecological validity and ensuring that it produces robust, reliable results across a wider variety of scenarios, including longer conversations [37,38] with systematically varied wording and formats, and larger sets of user-agents. Future work must check for and ensure broad coverage of relevant real-world communication patterns (eg, implicit vs explicit suicidal ideation, refusal to engage with resources, and ambivalence). Leveraging chatbot conversations with real users has the potential to both strengthen our evaluation of user-agent realism and further support VERA-MH validity by showing consistent scoring across both real and simulated conversations, as well as in datasets with more realistic (ie, less enriched) base rates of suicide risk. Additional human validation work using external clinicians would determine whether these reliability results generalize to new, external raters. Future VERA-MH expansions to other areas of risk, for example, harm to or from others, will also require human validation. Regarding future work on the rubric, the current 5 dimensions, which each encompass multiple indicators, could be divided into subdimensions to increase granularity. Future releases may modify the evaluation process so that each indicator mapping onto “high potential for harm” and “suboptimal” ratings is always rated (vs the current skip-out logic) for enhanced actionability for developers. There is also room for refinement of rubric definitions to further strengthen rating consistency; moving forward, the VERA-MH rubric criteria and rating process will continue to be closely informed by input from users and other stakeholders. Finally, given that VERA-MH is intended to apply to general-purpose and mental health-specific chatbots, future work may require distinct rubric versions for tools used in specific contexts; for example, wellness tools vs regulated devices and clinician-in-the-loop vs fully automated systems.

Conclusions

Generative AI has immense potential to increase access to mental health support at scale, but before such benefits can be realized, the core principle of nonmaleficence in health care must be prioritized—first, do no harm. This study found that the automated VERA-MH benchmark of AI safety for suicide risk detection and response exhibits strong alignment with a reliable consensus-based clinical reference. User-agent realism and the extent to which intended user-agent characteristics were accurately reflected, however, were more mixed. As these findings refer to an earlier version of VERA-MH and the evaluation continues to evolve within the rapidly changing AI landscape, validating subsequent releases will be an important next step. VERA-MH has the potential to help ensure that consumers and stakeholders are informed about AI safety in mental health and provide actionable guidance on how to make existing tools safer.